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PREFACE 


Management  Consulting  &  Research,  Inc.  (MCR)  provided 
support  to  the  Office  of  the  Assistant  Secretary  of  Defense 
(OASD )  for  Manpower,  Installations  and  Logistics  (MI&L)  under 
contract  number  MDA903-82-C-0278  for  the  examination  of  skill 
training.  MCR  analyses  will  assist  in  the  evaluation  and  support 
of  Service  training  programs. 

This  technical  report  is  a  contract  deliverable  that 
documents  the  skill  training  analyses  conducted  during  this 
project  to  determine  the  linkage  between  installation  level 
training  and  maintenance  productivity,  using  data  from  the  F-16 
aircraft.  We  were  able  to  determine  trends  and  establish 
relationships  between  Air  Force  Field  Training  Detachment  ( FTD ) 
training  and  maintenance  productivity.  We  also  determined  that 
the  use  of  simulators  for  training,  instead  of  real  aircraft,  did 
not  degrade  the  maintenance  productivity  of  units.  Thus,  it  can 
be  shown  that  simulators  provided  a  cost  savings  in  training. 

We  would  like  to  acknowledge  the  continuing  guidance  and 
assistance  of  the  members  of  the  Training  and  Education 
Directorate,  particularly  Mr.  Gary  Boycan,  Mr.  Michael  J. 
Kendall,  and  Lt.  Col.  R.  P.  Tyndell.  Also,  the  assistance  and 
cooperation  of  the  Air  Staff  and  members  of  the  Tactical  Air 
Command  (TAC)  made  our  work  possible.  ]  -pi — i — r — r 


!  m  * 

I  S'* 

‘  o  o 
O  > 

*0  «H 

>  s  g  a 

I  o  «H  O 

(*H  r"* 

G, 

3  43  «  S/J 

ft!  > 

-«  ^ _ 

\  *•>  <0 

J  «  >  4J 

r*  ^ 


EXECUTIVE  SUMMARY 


This  summary  includes  the  study  purpose,  organization  of 
this  report,  results  and  observations. 

A.  STUDY  PURPOSE  AND  REPORT  ORGANIZATION 

The  purpose  of  this  study  was  to  analyze  individual  skill 

training  conducted  at  the  installation  level  and  determine  the 

linkage  between  this  installation  level  training  and  maintenance 
"pit 

productivity.  used  units  containing  the  Air  Force  F-16 

aircraft  for  our  sample  since  it  is  one  of  our  newest  airplanes 

and  it  required  extensive  training/retraining  of  maintenance 

personnel.  The  use  of  simulated  aircraft  maintenance  trainers, 

rather  than  actual  aircraft  for  training,  was  specifically 

included  in  our  analysis  both  in  terms  of  training  capability  and 
TU*" 

cost  savings.  Our  task  included  these  subtasks: 

Jjv  })  identify  measures  of  maintenance  productivity; 

<2)  identify  training  status  of  organizations  and  in- 
/  dividuals,' 


qj9  -j)  reexamine  and  develop  analytical  methods  for  linking 
1  training  and  productivity; 

par  if)  analyze  information  and  evaluate  results;  and 

gja  ^document  the  results  of  the  analysis.  <c - - 


This  report  contains  three  sections. 


Section  I, 


Introduction,  describes  the  purpose,  background,  approach,  and 
organization  of  this  report.  Section  II,  Skill  Training 
Evaluation,  describes  Air  Force  maintenance.  Air  Force  training. 


research  background,  and  methodology.  Section  III,  Application 


of  Methodology  and  Results,  describes  the  application  of  our 
methodology,  results,  and  observations. 

B.  RESULTS  AND  OBSERVATIONS 

As  a  result  of  our  analysis  of  Air  Force  maintenance  and 
training  in  five  wings  we  determined  that  training  had  a 
statistically  significant  effect  on  maintenance  productivity.  We 
examined  F-16  aircraft  maintenance  and  training  for  three  major 

e 

maintenance  areas  and  the  related  skills:  flight  controls,  jet 
engine,  and  electrical  systems.  We  looked  for  trends  using 
graphical  techniques  and  then  applied  the  analysis  of  variance 
(ANOVA)  statistical  test. 

In  order  to  look  at  the  effects  of  the  use  of  aircraft 
simulators  we  compared  two  wings  that  had  been  trained 
differently:  one  with  simulators  and  one  without.  Also,  we 
found  that  the  use  of  simulators  required  fewer  aircraft  in 
support  of  maintenance  training. 

Our  overall  intention  of  showing  a  relationship  of  training 
to  maintenance  productivity  was  successful  and  provides 
quantified  data  to  support  the  conclusion  that  installation  level 
training  increases  productivity.  The  graphic  representations  of 
frequency  (number  of  work  actions  taken)  and  level  of  training 
versus  productivity  lead  us  to  the  conclusion  that  there  is  a 
positive  relationship  between  amount  of  training  and  productivity 
at  the  installation  level.  The  statistical  results  using  ANOVA 
indicate  that  for  jet  engine  maintenance  and  its  related  skill, 
as  well  as  for  electrical  systems  maintenance  and  its  related 


skill,  our  findings  are  statistically  significant  at  the  90 
percent  confidence  level.  We  also  found  a  positive  relationship 
for  flight  control  maintenance,  and  its  related  skill,  but  at  a 
lower  confidence  level. 

The  wing  that  was  trained  using  simulators  at  Luke  AFB 
performed  maintenance  as  productively  as  the  wing  at  Nellis  AFB 
that  was  trained  without  simulators.  This  was  accomplished  with 
a  cost  avoidance  of  approximately  one-half  aircraft  assigned  to 
support  training  throughout  the  period  observed.  This  cost 
avoidance  shows  that  use  of  simulators  for  training  can  save 
money  and  also  contribute  to  readiness  by  allowing  additional 
aircraft  to  be  assigned  to  operational  purposes. 

It  is  apparent  from  this  research  that  the  interim  results 
reported  previously  have  been  reinforced  by  our  current  analysis. 
It  is  obvious  that  training  does  directly  influence  maintenance 
and  that  delaying  the  training  of  new  maintenance  personnel  is 
detrimental  to  productivity.  The  wing  analysis  and  training 
sections  are  not  presently  attempting  this  linkage  of  training 
and  maintenance  since  their  primary  concerns  are  to  improve 
operational  flying  capability.  Thus,  data  is  collected  on 
maintenance,  and  training  is  conducted,  but  the  two  are  not  being 
correlated.  If  wing  headquarters  personnel  were  able  to  do 
analyses  similar  to  ours,  they  should  be  able  to  show  where 
training  needs  reinforcement  so  as  to  positively  affect 
maintenance. 

Some  improvements  to  maintenance  data  collection  could 
benefit  the  current  system  and  assist  analysis  at  the  wing  level. 


EXHIBIT 

II-l 

II-2 

II-3 

II-4 

II-5 

III-l 

III-2 

III-3 

III-4 

III-5 

III-6 

III-7 

III-8 

III-9 

III-10 

III-ll 

III-12 

III-13 

III-14 

III-15 


LIST  OF  EXHIBITS 


Maintenance  Data  Collection  Record . 

Typical  Tactical  Wing  Organization . 

Typical  Training  Path  of  Air  Force  Main¬ 
tenance  Personnel  ....  . 

Air  Training  Command-Field  Training  Detach¬ 
ment  Relationship  .  .  . 

Sample  ANOVA  Matrix  . 

Aggregate  Average  Productivity  and  Frequency 
by  Action  Code  (WUC  14000  ) . 

Training:  Wing-to-Wing  (WUC  14000)  . 

ANOVA  Matrix  For  WUC  14000 . 

Aggregate  Frequency  Effects  (WUC  14000)  .  .  . 

Aggregate  Training  Effects  (WUC  14000).  .  .  . 

Aggregate  Average  Productivity  and  Frequency 
by  Action  Code  (WUC  23000  ) . 

Training:  Wing-to-Wing  (WUC  23000)  . 

ANOVA  Matrix  for  WUC  23000 . 

Aggregate  Frequency  Effects  (WUC  23000)  .  .  . 

Aggregate  Training  Effects  (WUC  23000).  .  .  . 

Aggregate  Average  Productivity  and  Frequency 
by  Action  Code  (WUC  42000).  ...  . 

Training:  Wing-to-Wing  (WUC  42000)  . 

ANOVA  Matrix  for  WUC  42000 . 

Aggregate  Frequency  Effects  (WUC  42000)  .  .  . 

Aggregate  Training  Effects  (WUC  42000).  .  .  . 


vii 


PAGE 

II-3 

II-5 

II-7 

11-10 

II- 25 

II I - 4 
III-5 
III-6 
III-8 
II I -9 

III-10 

III-ll 

III-12 

III-14 

III-15 

.11-16 
I 11-17 
III-18 
III-19 
III-20 


LIST  OF  EXHIBITS  (CONT'D) 


EXHIBIT  PAGE 

III-16  ANOVA  Table  for  WUC  14000  .  III-23 

III-17  ANOVA  Table  for  WUC  23000  .  III-24 

II 1-18  ANOVA  Table  for  WUC  42000  .  III-25 

III-19  ANOVA  Matrix  for  Luke  AFB  and  Nellis  AFB  for 

Combined  WUCs  . . III-29 

I 11-20  ANOVA  Table  for  Combined  WUCs  for  Luke  AFB 

and  Nellis  AFB .  III-30 


I .  INTRODUCTION 


This  section  discusses  the  following: 

•  Purpose, 

•  Background, 

•  Approach,  and 

•  Organization  of  this  Report. 

A.  PURPOSE 

The  purpose  of  our  study  was  to  analyze  the  linkage  between 
installation  level  training  and  maintenance  productivity. 
Specifically,  we  concentrated  on  Air  Force  F-16  wings.  The  F-16 
aircraft  is  the  newest  addition  to  the  Air  Force  tactical 
inventory  and  the  first  to  use  simulated  aircraft  maintenance 
trainers  (SAMTs),  or  simulators,  for  training  mechanics.  This  is 
important  to  our  analysis  since  it  allowed  comparison  of  newly 
converted  wings  to  wings  converted  at  an  earlier  period.  The 
intent  was  to  collect  information  from  five  wings  located  in  the 
continental  United  States  (CONUS)  for  ease  of  data  collection  and 
to  compare  this  data  in  our  linkage  determination.  The  task 
consisted  of  five  subtasks: 

•  identify  measures  of  maintenance  productivity, 

•  identify  training  status  of  organizations  and 
individuals, 

•  reexamine  and  develop  analytical  methods  for  linking 
training  and  productivity, 

•  analyze  information  and  evaluate  results,  and 

•  document  the  results  of  the  analyses. 


B 


BACKGROUND 


OASD  (MI&L )  has  issued  policy  guidance  that  manpower  impacts 

will  be  considered  in  system  design  and  operation.  This 

consideration  requires  both  the  Office  of  the  Secretary  of 

Defense  (OSD)  and  the  Services  to  develop  improved  methods  for 

evaluating  manpower,  personnel,  and  training.  The  analysis  of 

critical  skills,  particularly  the  impact  of  required  level  of 

maintenance  skills  on  quality  of  maintenance,  is  a  key  factor  in 

these  evaluations.  Individuals  who  are  required  to  operate  and 

maintain  weapons  systems  reach  the  journeyman  level  through 

individual  training  conducted  at  the  installation  level. 

Due  to  a  growing  concern  for  the  problems  in  maintaining  and 

operating  existing  weapons  systems,  a  comprehensive  review  of 

training  was  initiated  by  OASD  (MI&L).  Some  initial  work  has 

been  completed  concerning  how  weapons  systems  support  can  be 

1/ 

improved  with  better  training.  In  their  first  report""'  MI&L 

describes  existing  on-the-job  training  (OJT)  programs  and 

proposes  ways  to  enhance  training,  especially  OJT,  in  order  to 

2/ 

improve  equipment  maintenance.  In  a  later  report""',  MI&L 

provides  an  overview  of  the  systems  developed  to  train 

individuals  in  selected  maintenance  skills.  The  report  covers 

the  entire  training  plan,  from  initial  skill  training  in  formal 

schools  to  individual  skill  training  received  in  the  unit.  Based 

on  these  completed  and  on-going  study  efforts,  and  in  support  of 

i/  Report  on  the  OJT  Study  Task:  Qn-The  Job  Training  in  the 

Department  of?  Defense,  OASD  ( MRA&L )  under  the  auspices  of  the 
Defense  Education  and  Training  Executive  Committee,  January  1981 
2/  Report  on  Individual  Skill  Training:  Maintenance  Training  in 
the  Department  of  Defense,  OASD  (MRA&L),  May  1982. 


that  is  not  easily  measured,  since  training  will  occur  whether 
the  simulator  is  available  or  not.  The  simulator  does  allow  the 
student  to  diagnose  rare  problems  and  thus,  on  the  occasion  that 
something  out  of  the  ordinary  occurs,  be  able  to  fix  it  properly. 
The  simulator  appears  to  provide  a  means  of  achieving  a  higher 
quality  of  maintenance,  although  our  method  of  measuring 
productivity  by  time-to-repair  will  not  make  this  readily 
obvious.  However,  our  intent  was  to  show  that  the  simulator 
provides  training  equal  to  that  obtained  using  a  real  aircraft. 
Thus,  the  aircraft  are  freed  for  operational  use  at  a  cost 
savings  to  the  Air  Force. 

In  our  analysis  of  the  linkage  of  unit  level  skill  training 
and  productivity,  our  focus  was  on  maintenance.  The  approach  we 
followed  was  to  choose  specific,  highly  visible  skills  and  to 
examine  the  training  provided  and  the  related  maintenance  tasks 
performed.  This  required  an  initial  research  effort  to  find 
appropriate  skills  that  required  installat ion-level  training  or 
for  which  training  was  provided.  Also,  the  initial  research 
effort  focused  on  available  sources  of  maintenance  measures  of 
productivity.  The  intent  of  our  analytical  effort  was  to 
establish  specific  quantitative  linkages  between  skill  training 
provided  by  units  and  maintenance  productivity. 

The  approach  we  used  relates  training  received  by 
maintenance  personnel  at  the  installation  level  to  on-the-job 
performance.  This  approach  includes: 

•  identifying  skills  taught  using  maintenance  simulators 
at  the  wing  level,  and  . 


•  developing  methods  for  analyzing  the  impact  of  these 
learned  skills  on  maintenance  productivity. 

The  research  focused  on  maintenance  skills  critical  to  the 
operation  of  the  F-16  aircraft.  The  study  was  conducted  in  the 
following  order: 

•  Select  appropriate  sample  organizations; 

•  Make  visits  to  selected  operational  units  and  their 
supporting  FTDs  to  examine  unit  training  and 
maintenance; 

•  Request  and  receive  training  status  and  maintenance 
productivity  information  from  the  units  selected; 

•  Research  the  use  of  simulated  aircraft  maintenance 
trainers  (SAMTs)  by  FTDs  for  training  maintenance 
personnel  at  the  wing  level; 

•  Reexamine  and  develop  methods  for  analyzing  the  data; 

•  Analyze  the  data;  and 

•  Document  the  results. 

D.  ORGANIZATION  OF  THIS  REPORT 

Following  this  introduction  are  two  other  sections  of  the 
report: 

•  Skill  Training  Evaluation,  and 

•  Application  of  Methodology  and  Results. 

Section  II,  Skill  Training  Evaluation,  discusses  Air  Force 
maintenance.  Air  Force  training,  research  background,  and 
methodology. 

Section  III,  Application  of  Methodology  and  Results, 
discusses  the  application  of  our  methodology,  results,  and 
observations . 


Supporting  information  is  presented  in  three  appendices: 
e  Reference  Sources, 
e  Backup  Data,  and 
e  ■  Abbreviations. 
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II.  SKILL  TRAINING  EVALUATION 


In  our  previous  research  we  set  out  to  establish  a  linkage 

between  individual  training  an  (^maintenance  productivity.  The 

results  o£  that  research  showed  definite  quantitative 

4  / 

relationships  could  be  establ  ished '  We  used  individual 
maintenance  personnel  training  information  maintained  by  the  wing 
maintenance  training  section  within  the  training  subsystem  of  the 
Maintenance  Management  Information  and  Control  System  (MMICS). 
Maintenance  productivity  was  measured  using  the  number  of  work 
hours  needed  to  complete  a  maintenance  action.  This  information 
was  extracted  from  the  Maintenance  Data  Collection  (MDC )  system 
based  on  information  routinely  entered  by  work  centers  performing 
maintenance.  These  same  sources  were  used  in  this  phase  of  our 
work  with  improved  results  using  increased  sample  size  and 
additional  analytical  evaluations. 

This  section  describes  the  following  topics: 
e  Air  Force  Maintenance, 

e  Air  Force  Training, 

e  Research  Approach,  and 

e  Methodology. 

A.  AIR  FORCE  MAINTENANCE 

In  this  section  we  will  discuss  Air  Force  maintenance  from 
the  perspective  of  information  availability  and  where  it  is 
performed  organizationally.  The  following  subsections  discuss: 


i/  Ibid 


Maintenance  Data  Collection,  and 


•  Maintenance  Organization. 

1.  Maintenance  Data  Collection 


The  Air  Force  maintenance  system  is  described  in  AFR 


5/ 

66-1  '  and  documents  a  sufficient  amount  of  information  to 
ascertain  what  was  worked  on,  what  was  wrong,  what  was  done,  and 
who  did  it.  The  mechanic  or  supervisor  prepares  a  Maintenance 
Data  Collection  Record  (AFTO  Form  349)  which  is  used  to  record 
data  for  the  MDC  system. 

An  example  of  this  form  is  shown  on  Exhibit  II-l.  "The 
Maintenance  Data  Collection  Record  is  used  to  record  all 
maintenance  in  the  66-1/66-5  systems.  Equipment  being  maintained 
is  identified  by  the  values  of  the  work  unit  code  (WUC)  and 

the  SRD  .  ( SRD,  or  Standard  Reporting  Designator,  is  a  three 

character  code  that  denotes  the  type  of  equipment  end- item. )  What 
is  wrong  with  the  equipment  is  given  by  the  how-malfunction  code 
©  .  What  was  done  in  the  course  of  maintenance  is  contained  in 
the  type-maintenance  ©  and  action-taken  ©  codes.  Who 
performed  the  maintenance  task  is  identified  through  the 
combination  of  work  center  and  the  maintenance  organization 
containing  the  work  center,  where  the  identification  of  the 
organization  is  provided  by  the  processing  routines  that  transfer 
the  hardcopy  information  to  MDC  records.  Man-hours  expended  are 
calculated  during  ADP  processing  from  the  start-hour,  stop-hour, 
and  crew  size  values  contained  in  .  A  separate  line  of 


5 /  Equipment  Maintenance:  Maintenance  Management,  Air  Force 

Regulation  (AFR)  6&-1,  Headquarters,  U.S.  Air  Force,  2  July  1980, 


MAINTENANCE  DATA  COLLECTION  RECORD 


MAINTENANCE  DATA  COLLECTION  RECORD 


information  is  recorded  each  time  there  is  a  change  in  any  of  the 
columns  A  through  N,  including  changes  in  crew  size  or 


composition,  and  each  line  results  in  a  distinct  record.  Employee 

number  ©  typically  names  a  shop  supervisor  or  lead  technician 

and  should  not  be  a  reliable  guide  to  identifying  individuals 

actually  performing  maintenance;  in  addition,  this  information  is 

6/ 

not  transferred  to  MDC  records."^ 

The  improved  F-16  Consolidated  Data  System  (CDS)  uses 
the  same  AFTO  Form  349  as  input  but  allows  for  supervisor 
comments  to  be  added.  The  advantage  of  CDS  over  MDC  is  the 
ability  to  aggregate  maintenance  information  in  a  more  useable 
form  and  with  flexibility  as  to  what  data  is  displayed.  The 
obvious  information  needed  is  what  was  worked  on  (work  unit  code 
or  WUC),  what  was  done  (action  taken  code),  how  many  workhours 
were  used,  and  who  did  the  work  (work  center). 

2.  Maintenance  Organization 

Organizationally,  each  wing  contains  both  flying 
squadrons  and  maintenance  squadrons.  Exhibit  II-2  shows  the 
typical  wing  organization.  It  should  be  noted  that  within  the 
Aircraft  Generation  Squadron  there  is  an  Aircraft  Maintenance 
Unit  (AMU)  for  each  flying  squadron.  The  AMU  provides 
organizational  maintenance  support.  The  other  maintenance 
squadrons  provide  intermediate  maintenance  support. 


1/  J.  Orlansky  and  J.  String,  Evaluating  the  Effectiveness  of  Main 
tenance  Training  by  Using  Currently  Available  Maintenance  Data, 
IDA  Paper  P-1574,  Institute  for  Defense  Analyses,  August  1981. 


Our  research  focused  on  FTDs  assigned  to  the  Tactical 
Branch  which  supports  TAC.  Exhibit  II-2,  shows  the  relationship 
between  the  maintenance  structure  of  a  tactical  wing  and  its 
supporting  FTD .  Typically,  there  are  three  or  four  flying 
squadrons  in  a  wing.  The  Deputy  Commander  for  Maintenance  (DCM) 
manages  equipment  maintenance  for  these  flying  squadrons.  The 
DCM  is  also  responsible  for  assuring  that  maintenance  training 
(FTD,  OJT,  and  informal  programs)  is  effective.  Maintenance  work 
is  performed  by  work  centers  in  the  Aircraft  Generation  Squadron 
(AGS),  Component  Repair  Squadron  (CRS),  and  Equipment  Maintenance 
Squadron  (EMS).  Some  work  centers  are  manned  primarily  by 
personnel  trained  in  a  single  skill  area  (e.g.,  all  jet  engine 
mechanics),  while  others  are  manned  by  personnel  trained  in 
several  skill  areas  (e.g.,  integrated  avionics  specialists, 
aircraft  electrical  systems  specialists,  and  jet  engine 
mechanics).  Team  maintenance  (two  or  more  individuals  performing 
a  single  maintenance  task)  is  standard  procedure  in  most  wings. 
Cross-skill  maintenance  (personnel  who  are  trained  in  one  skill 
area  performing  a  maintenance  task  associated  with  a  different 
skill  area)  is  performed  as  required  by  workload  and  personnel 
shortages . 

B.  AIR  FORCE  TRAINING 

The  focus  of  this  study  is  the  impact  of  training  conducted 
at  the  installation  level  on  maintenance  job  performance.  The 
typical  training  flow  for  Air  Force  maintenance  personnel  is 
shown  on  Exhibit  II-3.  These  enlistees  attend  initial  entry 
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training  (basic  military  training  and  technical  school)  prior  to 
assignment.  During  technical  school,  they  will  learn  the  basic 
skills  required  within  their  Air  Force  Specialty  Code  (AFSC). 
Next,  they  receive  equipment-specific  training  in  an  installation 
Field  Training  Detachment  (FTD) .  Much  of  this  FTD  training  uses 
simulators.  This  is  followed  by  on-the-job  training  (OJT)  in  the 
unit . 

The  following  subsections  discuss: 

•  Installation  Training,  and 

•  Aircraft  Maintenance  Simulators. 

1 .  Installation  Training 

The  primary  functions  of  an  installation  school  include 
the  following: 

•  to  supplement  the  generalized  training  received  in 
formal  schools  with  equipment-specific  training  related 
to  types  of  equipment  located  at  that  installation; 

•  to  provide  transition  training  to  personnel  whose 
previous  experience  has  been  on  other  models  of 
equipment;  and 

•  to  provide  upgraded  and  refresher  training. 

There  are  two  types  of  installation  schools:  officially 
recognized  schools  and  unofficial  schools.  Officially  recognized 
schools  are  specifically  funded  and  are  manned  in  accordance  with 
authorization  documents.  Although  these  schools  belong  to  the 
training  command,  their  main  purpose  is  to  meet  the  specific 
needs  of  the  associated  unit  at  the  installation.  Included  among 
the  officially  recognized  schools  are  the  Air  Force  Field 
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Training  Detachments,  which  focus  on  aviation  maintenance 
training. 


Unofficial  installation  schools  are  established  by  an 
operational  unit,  using  its  own  manpower  and  funding  resources. 
In  the  Air  Force,  unofficial  schools  are  used  extensively  in  the 
Tactical  Air  Command  (TAC).  The  unofficial  schools  in  TAC 
provide  training  in  addition  to  that  provided  by  the  FTDs. 

Most  of  the  installation  school  training  in  the  Air 
Force  takes  place  at  FTDs.  Exhibit  II-4  shows  the  organizational 
relationship  between  the  Air  Training  Command  and  the  FTDs. 

2 .  Aircraft  Maintenance  Simulators 

The  use  of  aircraft  maintenance  simulators  can  be  very 
helpful  in  conducting  skill  training.  This  section  addresses 
these  topics:  the  effects  of  having  maintenance  simulators,  a 
description  of  what  kind  of  simulators  are  available  for  F-16 
training  at  the  wing  level,  and  what  the  simulators  are  used  for. 
a .  Effects  of  Having  Maintenance  Simulators 

As  a  training  device,  a  maintenance  simulator  can 
be  designed  to  provide  facilities  important  for  instructing 
students,  in  contrast  to  actual  equipment  that  is  designed  to 
operate  effectively  in  an  operational  environment.  Maintenance 
simulators  can  be  designed  to  include  a  large  variety  of  faults 
with  which  maintenance  personnel  should  be  familiar,  including 
faults  that  cannot  be  demonstrated  conveniently  on  actual 
equipment  trainers  or  faults  that  occur  rarely  in  real  life. 


HEADQUARTERS  (HQ) 
AIR  TRAINING  COMMAND 


FIELD  TRAINING  DETACHMENT  RELATIONSHIP 


All  modern  maintenance  simulators  incorporate  some 
type  of  computer  support.  This  support  provides  numerous 
training  enhancements. 

•  The  computer  can  automatically  record  student  responses 
to  training  situations,  thereby  reducing  the  need  for 
constant  observation  by  the  instructor  (and  providing 
accurate  records  of  performance). 

•  The  computer  can  assist  students  without  an 
instructor's  intervention. 

•  The  instructors  can  use  information  collected  by  the 
computer  to  better  guide  each  student  through  training 
by  focusing  on  weak  areas. 

Simulators  can  be  made  rugged  enough  to  sustain 
damage  or  abuse  by  students  and  thus  provide  greater  reliability 
and  availability  in  the  classroom  than  is  often  possible  with 
actual  equipment.  Training  which  would  be  avoided  because  of 
safety  reasons  (e.g.,  exposure  of  students  to  dangerous 
electrical  charges  or  hydraulic  pressures)  can  be  undertaken  with 
little  risk  on  a  simulator. 

F-16  maintenance  training  conducted  at  the  FTD 
level  is  enhanced  by  the  use  of  aircraft  simulators.  These 
simulators  are  organized  into  sets.  A  full  set  may  have  11 
different  simulators  or  trainers.  Each  individual  trainer 
simulates  different  functions  aboard  the  aircraft.  These 
simulators  are  divided  into  two  major  types: 

•  Simulated  Aircraft  Maintenance  Trainers  (SAMTs).  These 
computerized,  fault-oriented  trainers  provide  an 
advanced  method  of  training  maintenance  technicians  to 
diagnose  and  correct  system  malfunctions.  SAMTs 
provide  a  method  of  teaching  operational  procedures  in 
a  very  cost-effective  manner.  Power  requirements  are 
lower  than  with  actual  equipment,  and  simulator  system 
response  to  student  inputs  is  identical  to  that  of  the 
aircraft. 


•  Hardware  Trainers.  These  trainers  provide  a  physical 

simulation  of  actual  maintenance  areas.  These 
simulators  focus  on  "hands-on"  training  experiences  for 
the  repair  of  common  malfunctions  in  some  of  the  more 
mechanical  areas  of  the  aircraft;  diagnostic  skills  are 
not  stressed  within  these  simulators. 

b.  P-16  Aircraft  Maintenance  Simulators 


The  full  group  of  simulators  is  called  a  mobile 
training  set  (MTS)  and  consists  of  seven  SAMTs  and  four  hardware 
simulators  (or  trainers).  Each  simulator  is  numbered,  although 
not  all  numbers  are  used.  Six  SAMTs  are  manufactured  by 
Honeywell  and  one  by  Educational  Computer  Corporation  (ECC).  The 
hardware  trainers  are  made  by  General  Dynamics.  The  list  below 
shows  simulator  number,  title  and  manufacturer: 


Simulator 

Number  Simulator  Title  and  Manufacturer 


2 

3 

4 
6 

7* 

10 

11 

12 

13 

14 

15 
22 


SAMT — Flight  Control/lnstrumentation  (Honeywell) 
SAMT — Navigation  (Honeywell) 

SAMT— Electronics  (Honeywell) 

Hardware — Seat  and  Canopy  (General  Dynamics) 

SAMT — Hydraulics  (Honeywell) 

SAMT — Engine  Start  (Honeywell) 

SAMT — Engine  Diagnostics  (Honeywell) 

SAMT — Engine  Operating  Procedure  (Honeywell) 
Hardware — F-100  Engine  (General  Dynamics) 
Hardware — Gun  (General  Dynamics) 

Hardware — Fuel  (General  Dynamics) 

SAMT — Environmental  Control  (ECC) 


"No  longer  in  use. 


c.  Use  of  Simulators 

The  following  listing  shows  the  simulator  number 
and  title,  aircraft  equipment  repairs  by  primary  work  unit  codes 
(WUCs)  taught  using  the  simulators,  and  the  course  title/AFSC  for 
which  the  simulator  is  used. 
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Simulator 

2,  Plight  Control/ 
Ins trumen tat ion 


3,  Navigation 


WUC-- Aircraft 
Equipment 

14A00,  Primary  Flight 
Control  Electronics 
14B00,  Primary  Flight 
Control  Actuators 
51A00,  Primary  Flight 
Instruments 
51F00,  Control  Air 
Data  Computer 

71A00,  TACAN  Navigation 
Set 

71B00,  Instrument 
Landing  Set 


4,  Electronics 


42000,  Electrical 
Power  Supply 


6,  Seat  and  Canopy 


12000,  Crew  Station 
System 


10,  Engine  Start 


11,  Engine  Diagnos¬ 
tics 

12,  Engine  Operat¬ 
ing  Procedure 

13,  F-100  Engine 

14,  Gun 


23000,  Turbofan 
Power  Plant; 
v24000.  Auxiliary 
Power  Plant 


75A00,  Gun  System 


15,  Fuel 


46000,  Fuel  System 


22,  Environmental 
Control 


41000,  Environmental 
Control  System 


Courses — AFSC 


Integrated  Avionics 
Instrument  and  Flight 
Control  System 
Specialist  (F-16); 
for  AFSC  326X7. 


Integrated  Avionics 
Navigation,  and  Pen¬ 
etration  Aids  Systems 
Specialists  (F-16);  for 
AFSC  326X8. 

Aircraft  Electrical 
Systems  Technician 
(F-16);  for  AFSC  423X0. 

Aircrew  Egress  Systems 
Technician  (F-16);  for 
AFSC  423X2. 


Jet  Engine  Technician 
(F-16);  for  AFSC  426X4. 


Weapons  System  Maintenance 
Technician  (F-16);  for 
AFSC  462X0. 

Aircraft  Fuel  Systems 
Technician  (F-16);  for 
AFSC  423X3. 

Aircraft  Environmental 
System  Technician  (F-16); 
for  AFSC  423X1. 


C.  RESEARCH  BACKGROUND 

This  study  examines  and  quantifies  the  impact  of  FTD 
training,  particularly  training  that  uses  maintenance  simulators, 
on  maintenance  productivity.  We  relate  efforts,  measured  in 
workhours  to  complete  a  maintenance  action,  to  work  center 
training  status. 

Many  factors,  beyond  training  at  the  unit/base  level  and  the 
use  of  actual  or  simulated  equipment,  can  profoundly  influence 
our  capability  to  maintain  military  equipment.  These  factors 
include: 

•  quality  of  personnel  recruited  by  the  Military 
Services, 

•  policies  used  by  the  Services  to  assign  recruits  to 
occupational  specialties, 

•  amount  and  type  of  training  to  be  accomplished  at 
technical  schools, 

•  complexity  of  the  information  that  must  be  assimilated 
in  order  to  accomplish  maintenance, 

•  quality  of  maintenance  supervision, 

•  equipment  design,  and 

•  maintenance  policy. 

These  factors  are  recognized  as  being  important 
considerations  in  equipment  maintenance  analyses,  but,  due  to  our 
specific  concern  in  this  study  on  training  and  maintenance 
simulators,  these  factors  are  not  explicitly  addressed.  The 
remainder  of  this  section  on  research  background  discusses: 

•  Assumptions, 

•  Analytical  Considerations, 

•  Supporting  Data,  and 

•  Other  Information. 
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1  *  Assumptions 

The  analytical  task  of  linking  training,  frequency,  and 
productivity  measures  is  based  on  several  major  assumptions 
relating  to  the  data  and  the  procedures  followed  to  process  the 
data.  These  assumptions  include  the  following: 

e  An  implicit  assumption  in  this  analysis  is  that  the 
data  being  analyzed  is  accurate  and  complete.  Problems 
with  data  recording  and  data  entry  could  lead  to  biased 
or  inaccurate  data  analyses.  In  spite  of  these 

possible  inaccuracies  in  the  data  collection  system, 
this  study  is  based  on  information  entered  in  the  MDC 
system.  Data  which  is  verifiable  as  inaccurate  is 
excluded  from  our  analysis.  All  other  data  is 

included. 

e  Data  entered  in  the  MDC  system  is  assumed  to  represent 
"successfully  completed"  work  actions. 

e  Time  of  performance  shown  on  MDC  printouts  for  work 
actions  is  the  elapsed  time  (difference  between  start 
and  stop  time).  Time  for  breaks,  meals,  etc.  is  not 
included. 

e  Each  member  of  a  work  center  has  an  equal  opportunity 
to  perform  a  particular  action.  Thus,  for  all  work 
performed,  the  calculations  will  use  the  total  workers 
(with  appropriate  AFSC)  in  each  work  center. 

e  We  can  identify  particular  WUCs  that  are  associated 
with  actions  taught  using  maintenance  simulators. 
However,  these  WUCs  are  often  very  specific.  In  order 
to  match  this  information  with  the  level  of  detail 
obtained  in  our  WUC-related  data  requests,  we  are 
assuming  that  associated  higher  level  WUCs  are  also 
taught,  in  sufficient  detail,  on  the  SAMTs. 

e  If  a  maintenance  simulator  is  supposed  to  be  used  in 
training,  and  an  MTS  is  present  at  a  base,  then  we  are 
assuming  that  the  SAMT  is  used  in  all  appropriate 
training  (we  are  not  accounting  for  any  simulator 
downtime ) . 

2.  Analytical  Considerations 

We  have  attempted  to  establish  a  relationship  between 
work  centers,  WUC,  and  installation-level  (FTD)  training  using 


standard  output  data  from  base-level  information  systems  (MDC  and 
MMICS).  We  have  tried  to  assess  training  effectiveness  measured 
by  the  time  documented  to  do  a  task.-  This  disregards  quality  of 
work  and  assumes  that  all  work  was  done  correctly.  Consequently, 
some  biases  may  be  incorporated  in  our  work.  A  few  are  listed 
here. 

e  The  task  description  for  each  WUC  is  very  general  for 
defining  similar  tasks.  Difficulties  encountered  in 
tasks  are  not  well  defined  by  such  descriptions  as 
"remove  and  replace." 

#  Our  interpretation  of  personnel  awaiting  PTD  training 
is  that  they  are  untrained.  Actually  these  personnel 
have  received  basic  technical  training  in  their  APSC 
and,  in  some  cases,  have  job  experience  on  other 
aircraft  doing  similar  work.  Because  of  this,  our 
comparison  of  work  centers  shows  large  differences  in 
numbers  of  untrained  personnel  and  only  small 
differences  in  time  to  perform  work.  Since  we  did  not 
know  which  personnel  in  the  work  center  did  a  specific 
job,  we  assumed  all  personnel  could  have  done  the  job. 
This  may  be  compensated  for  by  the  fact  that  we  used 
average  times  for  several  jobs  in  the  same  work 
center. 


e 


3. 


Record  keeping  for  time  spent  on  a  task  may  vary  from 
one  work  center  to  another.  Although  timekeeping  is 
important,  the  requirement  to  document  all  time  while 
at  work  may  disguise  how  much  time  was  actually  spent 
on  a  particular  job. 

Inter-wing  comparisons  may  be  biased  by  different  FTD 
training,  different  approaches  to  using  personnel,  and 
different  time-keeping  procedures. 

The  use  of  cross-training  in  mixed  (more  than  one  AFSC) 
work  centers  may  also  lead  to  biases  in  establishing 
the  WUC-training  relationship  in  this  methodology. 


Supporting  Data 


The  details  of  our  application  are  in  Section  III.  We 


used  information  from  the  five  wings  listed  below  (full  titles 
and  persons  interviewed  are  provided  in  Appendix  A): 


•  Hill  AFB,  UT  —  388th  TFW, 

•  Luke  AFB,  AZ  —  58th  TTW, 

•  MacDill  AFB,  FL  —  56th  TTW, 

•  Nellis  AFB,  NV  —  474th  TFW,  and 

•  Shaw  AFB,  SC  —  363rd  TFW. 

We  requested  six  months  of  maintenance  data  for  the 
period  January  through  June  1983.  This  assured  a  sufficiently 
large  amount  of  observations  to  support  our  analyses  of  these 
areas  of  maintenance:  jet  engine,  flight  control/instrumenta¬ 
tion,  and  electrical  power  supply.  We  had  a  total  of  3889 
useable  maintenance  actions  in  this  data  set.  This  allowed  us  to 
make  comparative  analyses  among  forty-six  work  centers  and  five 
wings.  Also,  we  made  comparisons  between  two  wings,  the  58th  TTW 
and  474th  TFW,  that  trained  in  a  different  manner — one  with 
simulators  and  one  without  simulators. 

We  received  training  information  for  all  work  centers 
included  in  our  analysis  for  the  same  six  month  period.  The 
training  data  was  for  each  individual  assigned,  aggregated  by 
work  center. 

4 .  Other  Information 

In  the  course  of  our  research  we  attempted  to  gather 
data  concerning  two  other  aspects  of  maintenance  and  training. 
The  first  was  the  repair  of  components  initially  determined  to  be 
faulty  and  later  determined  upon  retest  to  be  all  right.  The 
second  was  the  cost  of  SAMTs  and  the  relationship  to  aircraft 
used  for  training. 


b 


Cost  of  SAMTs  and  Aircraft 


In  order  to  perfor.  a  cost  comparison,  we 
requested  information  on  F-16  SAMT  costs.  The  reason  for  this  is 
that  we  found  that  wings  having  supporting  FTDs  with  a  full  set 
of  SAMTs  were  providing  approximately  one  half  less  aircraft  per 
month  for  training  support  than  wings  without  SAMTs.  Thus,  it 
appears  that  the  cost  of  a  SAMT  set  may  be  compared  with  one  half 
of  an  F-16  cost.  The  following  cost  figures  were  provided  by  the 
Air  Force  (AF/MPP).  The  simulator  title  and  relationship  to 
aircraft  equipment  and  skills  trained  are  shown  in  Section 
II.B.2.C.  These  costs  are  for  SAMTs  only  and  do  not  include 
hardware  trainers  which  all  wings  have. 

SAMT  Number  and  FY83  Cost  (?) 


2 

$ 

293,005 

3 

233,488 

4 

198,432 

10 

220, 150 

11 

313,039 

12 

286,162 

22 

130,000 

TOTAL 

51, 

674,276 

Aircraft  cost  for  the  F-16  for  FY83  is  planned  to 
be  $2,254,600,000  for  120  planes  or  $18,788,333  each."^  Thus, 
the  $1,674,276  SAMT  set  produces  a  net  cost  avoidance  of 
$7,719,891  for  one-half  aircraft  needed  for  maintenance  training. 


D.  METHODOLOGY 

The  methodology  used  for  examining  the  linkage  between 
FTD-provided  training,  specifically  that  training  using  SAMTs  and 


V  Casper  W.  Weinberger,  Annual  Report  to  the  Congress  by  DoD  for 
FY86 ,  Department  of  Defense,  February  1983,  p.  168. 
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hardware  trainers,  and  maintenance  productivity  relies  heavily  on 


the  work  MCR  accomplished  in  our  previous  research.  The 
inference  is  that  as  the  number  of  FTD-trained  personnel  in  a 
work  center  increases,  the  productivity  of  that  work  center  will 
increase.  Productivity  is  measured  in  terms  of  the  workhours 
used  to  complete  a  specific  work  action.  Obviously,  the 
complexity  of  work  actions  can  vary  even  when  a  specific 
subassembly  is  being  worked  on  and  the  action  code  used  is  fairly 
specific.  This  complexity  or  difference  in  work  performed  should 
be  statistically  smoothed  by  taking  a  large  enough  sample  of 
particular  types  of  work. 

The  following  subsections  will  address  three  subjects: 

•  Determination  of  Training  Status, 

•  Determination  of  Maintenance  Productivity,  and 

•  Analytical  Techniques. 

1 .  Determination  of  Training  Status 

The  process  of  determining  the  training  status  of 
specific  work  centers  involves  obtaining  the  training 
requirements  for  the  skills  involved  in  performing  the  work, 
finding  which  individuals  have  received  the  required  FTD  course 
training,  and  calculating  the  overall  work  center  training 
status.  The  procedure  we  followed  in  this  task  for  calculating 
trained/untrained  status  is  listed  below. 

•  Determine  which  skills  should  be  examined  based  on  our 
experience  in  our  prior  work. 

•  Request  course  status  summaries  from  each  wing  for  the 
skills  chosen  for  examination. 


•  Calculate  trained/untrained  personnel  status  for  any 
work  center  that  will  be  examined.  The  examination  of 
specific  work  centers  is  determined  by  analysis  of 
maintenance  productivity  information. 

•  The  determination  of  whether  a  person  is  trained  or 
untrained  proceeds  as  follows: 

If  a  person  has  taken  the  required  FTD  course  for 
his  AFSC,  he  is  considered  to  be  trained. 

If  a  person  has  not  taken  the  required  FTD  course 
for  his  AFSC,  he  is  considered  to  be  untrained. 

•  Calculate  work  center  trained  percentages  for  specific 
skills  ( AFSCs ) .  When  a  work  center  contains  only  one 
AFSC,  then  the  calculation  consists  of  dividing  the 
number  of  trained  personnel  by  the  total  number  of 
personnel  in  the  work  center.  When  a  work  center  has 
more  than  one  AFSC,  then  the  calculation  consists  of 
dividing  the  number  of  trained  personnel  in  the  AFSC 
that  pertains  to  the  work  performed  by  the  total  number 
of  personnel  having  that  particular  AFSC  in  the  work 
center. 

2.  Determination  of  Maintenance  Productivity 

The  procedure  for  determining  the  maintenance 
productivity  of  specific  work  centers  starts  with  the  choice  of 
specific  work  unit  codes  (WUCs)  to  be  examined.  This  was 
assisted  by  our  prior  research  in  which  we  determined  that  there 
were  three  categories  of  maintenance  with  related  skills  that 
would  be  of  interest:  jet  engines,  flight  control/ 

instrumentation,  and  electrical  systems. 

Having  determined  which  WUCs  were  to  be  examined,  a 
request  to  each  wing  for  a  specifically  formatted  report  for  a 
particular  time  period  resulted  in  a  large  amount  of  information 
for  use  in  our  analyses.  We  were  assisted  greatly  in  our  data 
request  and  formats  by  representatives  of  the  Tactical  Air 
Command  (TAC),  located  at  their  headquarters,  as  well  as  TAC 
personnel  at  the  wing  level. 
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Our  general  procedure  for  determining  work  center 
maintenance  productivity  using  the  output  data  from  the  MDC 
system  is  as  follows: 

•  Request  a  WUC  report  that  specifies  the  WUCs  required 
for  analysis.  The  time  period  requested  was  January 
through  June  1983.  The  report  was  requested  for  all 
work  centers  and  for  all  action  taken  codes. 

•  After  receipt  of  the  WUC  report  from  each  wing,  examine 
it  for  completeness.  Then  see  which  work  centers  have 
the  appropriate  AFSCs  for  the  WUCs  under  considera¬ 
tion. 

•  Separate  and  aggregate  the  WUCs  by  work  center  and 
action  taken  code.  Then  sum  to  get  total  hours  for 
each  work  center/action  taken  code.  At  this  point,  the 
analysis  of  maintenance  productivity  must  incorporate 
other  variables;  i.e.,  total  number  of  actions  taken  by 
each  work  center  (frequency)  and  level  of  work  center 
training.  Number  of  workers  and  training  information 
are  received  from  other  wing  data  bases. 

•  The  final  analytical  step  is  to  make  comparisons  among 
work  centers  to  see  what  effects  training  and  other 
measurable  variables,  in  this  instance  frequency,  have 
on  maintenance  productivity. 

3.  Analytical  Techniques 

The  term  analytical  techniques  means  the  integrated 
step-by-step  process  followed  in  analyzing  the  training  and 
maintenance  action  data  that  we  requested/received  from  five  Air 
Force  F-16  wings. 

We  used  the  following  systems  as  our  data  sources: 

•  the  MDC  system  and  related  CDS  for  F-16  maintenance, 
which  provided  maintenance  information,  and 

•  the  training  subsystem  of  the  Maintenance  Management 
Information  and  Control  System  (MMICS),  which  provided 
training  information. 

Special  terms  used  in  our  analysis  are  defined  below. 

•  Observation  is  a  single  completed  work  action  in  a  work 
center . 
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Frequency  is  the  total  number  of  work  center 
observations  (work  actions)  for  the  period  examined. 


•  Productivity  is  the  total  number  of  workhours  used  to 
complete  work  action  in  a  work  center. 

It  is  important  to  note  that  as  the  number  of  workhours 
increases  productivity  decreases.  This  is  because  we  chose 
the  standard  industrial  engineering  definition  for  productivity — 
less  time  is  better  since  it  is  the  total  time  required  to 
perform  a  task  or  produce  an  output.  The  alternative  definition 
sometimes  used  is  output  per  unit  of  time. 

We  used  two  techniques  for  examining  the  data: 

•  Examine  productivity  graphically  to  determine  trends: 

using  wing-to-wing  data  for  each  WUC  and  its 
associated  action  codes, 

using  aggregated  training  effects  for  each  WUC, 
and 

using  aggregated  frequency  effects  (number  of  work 
actions)  for  each  WUC. 

•  Use  analysis  of  variance  (ANOVA)  to  statistically  test 
our  sample  results. 

Each  of  these  is  discussed  below. 


Examine  Productivity  Graphically  to  Determine 
Trends 


This  technique  involves: 


•  Aggregating  the  workhours  for  a  specific  WUC  by  action 
taken  code  ("action  code")  for  each  wing.  Plotting 
this  information  and  examining  the  overall  relationship 
between  training  and  productivity. 

•  Aggregating  the  workhours  for  each  WUC  by  wing, 
separating  the  data  by  training  status  (percent 
trained)  and  frequency  (number  of  work  actions). 
Plotting  the  information  and  examining  for  aggregate 
training  effects  and  aggregate  frequency  effects. 


b 


Use  Of  ANOVA 


This  approach  involves  a  statistical  test  of 
sample  data  to  determine  whether  or  not  the  results  observed  are 
statistically  significant.  The  steps  in  this  technique  include: 

•  classification  of  all  work  center  observations  into 
"cells"  in  an  ANOVA  matrix,  as  shown  on  Exhibit  II-5; 

•  computation  of  average  productivity  figures  within  each 
ANOVA  matrix; 

•  computation  of  variation  statistics  attributable  to: 

-  training, 

frequency  (number  of  work  actions),  and 
the  error  term;  and 

•  performing  an  F-test  for  variable  significance. 

The  choice  of  low,  medium,  and  high 
classifications  is  dependent  upon  the  training  status  (trained 
percent)  of  work  centers  and  the  number  of  observations 
(frequency)  in  the  sample.  The  division  into  the  various 
classification  will  vary  from  one  WUC  to  another  since  training 
and  frequency  are  different.  Also  the  number  of  classifications 
may  be  increased  (e.g.,  low,  low-medium,  high-medium,  high)  if 
the  sample  data  allows  for  it  in  terms  of  sample  size  and 
distribution. 
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III.  APPLICATION  OF  METHODOLOGY  AND  RESULTS 


We  used  two  analytical  techniques  for  our  analysis.  The 
first  is  quite  elemental:  the  examination  of  graphical  trends.  A 
statistical  method  was  used  for  the  second — analysis  of  variance. 
This  section  discusses  the  following: 

•  Application  of  Methodology, 

•  Results,  and 

•  Observations  with  respect  to  the  results  that  were 
obtained. 

A.  APPLICATION  OF  METHODOLOGY 

We  chose  three  system  WUCs  for  our  analysis — each  of  which 
has  a  related  AFSC  or  skill.  This  choice  was  based  on  the 
experience  gained  in  our  previous  work  and  our  knowledge  that 
these  would  allow  for  a  representation  of  total  wing  maintenance 
in  key  areas.  These  WUCs  and  their  related  AFSCs  or  skills  are 
shown  below. 

WUC  Number  and  Title  AFSC  Number  and  Title 

14000,  Flight  Control  326X7,  Integrated  Avionics 

Instrument  and  Flight  Control* 
System  Specialist  (F-16)  * 

23000,  Turbofan  Power  Plant  426X4,  Jet  Engine  Technician 

(F-16) 

42000,  Electrical  Power  Supply  423X0,  Aircraft  Electrical 

Systems  Technician  (F-16) 

We  then  requested  workload  data  for  these  three  WUCs  from 
the  five  wings  and  at  the  same  time  requested  training  data  for 
the  three  associated  AFSCs.  This  data  was  aggregated  by  work 


center  and  WUC  actions.  The  aggregated  data  for  each  work 
center,  showing  average  workhours  per  observation  and  frequency 
is  provided  in  Appendix  B,  Backup  Data.  The  WUC  data  and 
associated  action  codes,  which  had  sufficient  observations  for 
our  analysis,  are  shown  below. 

_ WORK  UNIT  CODE _ 

14000  23000  42000  ACTION  CODE  TITLE 

G  Repair /Replace  Minor  Parts 

P  P  P  Removed 

Q  Installed 

R  R  R  Removed  and  Replaced 

T  T  Removed  for  Cannibalization 

U  U  Replaced  After  Cannibalization 

X  XX  Test-Inspection-Service 

Y  Y  Y  Troubleshoot 

A  statistical  test  was  needed  in  order  to  make  generaliza¬ 
tions  based  on  limited  data.  This  test  revealed  whether  there 
is  a  statistically  significant  difference  in  the  observed  mean 
productivity  results  as  opposed  to  the  observed  differences 
simply  being  due  to  chance.  One  technique  that  allowed  us  to 
examine  our  variables  is  called  analysis  of  variance  (ANOVA). 
ANOVA  tests  the  statistical  significance  of  sample  means  or  tests* 

T 

i 

the  null  hypothesis  that  the  sample  means  are  equal.  We  have 
performed  this  test  for  both  frequency  (number  of  work  actions) 
and  training  variables  to  determine  their  effect  on  productivity. 

4 

A  complete  description  of  the  ANOVA  technique  and  its  use,  as 
well  as  the  data  we  used,  are  provided  in  Appendix  B,  Backup 
Data. 

The  reminder  of  this  section  will  discuss  each  of  the  three 
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system  WUCs  we  examined  and  the  related  training  status  of  the 
performing  work  centers: 

•  Flight  Control, 

•  Turbofan  Power  Plant,  and 

•  Electrical  Power  Supply. 

1.  Flight  Control  (WUC  14000) 


The  examination  of  productivity  versus  training  status 

and  frequency  of  work  performance  required  the  calculation  of 

trained  percent  by  individual  work  center,  aggregated  by  wing' 

(base),  as  well  as  the  number  of  observations  (frequency)  for 

each  action  code.  These  data  are  shown  on  Exhibit  III-l  for  WUC 

14000,  which  relates  to  AFSC  326X7.  Similarly  productivity  by 

action  code  must  be  calculated;  this  is  also  shown  on  Exhibit 

III-l.  The  productivity  (in  workhours)  by  action  code  has  been 

plotted  and  connected  by  lines  on  Exhibit  III-2  for  each  wing 

(base).  Each  wing  has  a  different  amount  of  trained  people. 

This  exhibit  is  intended  to  shown  any  trend  that  may  exist 

between  training  and  productivity.  Training  effects  are  not 

readily  apparent.  For  example,  Hill  AFB ,  with  the  lowest  amount* 

• 

s 

of  trained  personnel  falls  near  the  middle  of  other  bases  with 
higher  amounts  of  trained  personnel.  Consequently,  the  use  of 
frequency  (number  of  work  actions)  must  be  introduced.  Exhibit 
III-3  combines  training  and  frequency  with  productivity  data. 
This  matrix  combines  all  action  code  productivity  data  and 
displays  training  and  frequency  into  general  groupings  of  low  to 
high.  This  same  matrix  will  be  used  for  ANOVA  calculations  later 
in  our  analysis. 
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Exhibit  III-2.  TRAINING:  WING-TO-WING  (WUC  14000) 


The  aggregate  frequency  effects  are  shown  on  Exhibit 
III-4  and  aggregate  training  effects  on  Exhibit  III-5.  The 
frequency  effects  are  ambiguous  and  do  not  provide  a  great  deal 
of  information.  Similarly,  the  training  effects  do  not  reveal 
any  particular  trend. 

The  other  analytical  technique  is  the  application  of 
ANOVA  to  our  data.  The  same  specific  effects  are  seen  in  our 
ANOVA  matrix  (Exhibit  in'-!).  The  values  go  up  and  down  moving 
from  low  to  high  training  or  frequency.  The  statistical  ANOVA 
results  are  discussed  in  Section  III.B.,  Results. 

2.  Turbofan  Power  Plant  (WUC  23000  ) 

As  with  the  other  system  WUCs  and  related  AFSCs,  the 
calculation  of  training  status  and  aggregation  of  productivity 
and  frequency  data  was  accomplished  for  WUC  23000  and  its  related 
AFSC  426X4.  This  information  is  shown  on  Exhibit  III-6.  The 
productivity  by  action  code  data  has  been  plotted  and  connected 
on  Exhibit  III-7  for  each  wing  (base)  so  as  to  show  any  trends 
that  might  be  apparent . 

Training  effects  in  this  instance  are  fairly  evident:, 
although  the  wing  training  status  does  not  vary  much,  we  do  see 
sizable  differences  in  productivity  between  the  highest  trained 
wings  and  the  lowest.  Exhibit  111-8  combines  training  and 
frequency  effects  with  productivity  data.  This  matrix  combines 
all  action  code  productivity  data  and  displays  training  and 
frequency  as  general  groupings  of  low  to  high.  We  have  used  four 
frequency  classifications  due  to  the  sample  size  (50  percent 
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Exhibit  III-4.  AGGREGATE  FREQUENCY  EFFECTS  (WUC  14000) 


(%  TRAINED) 


Exhibit  III-5 .  AGGREGATE  TRAINING  EFFECTS  (WUC  14000) 


ACTION  CODE  AVERAGE  PRODUCTIVITY  (WORKHOURS) 
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Exhibit  III-6 .  AGGREGATE  AVERAGE  PRODUCTIVITY  AND  FREQUENCY 

BY  ACTION  CODE  (WUC  23000) 


Exhibit.  HI-7.  TRAINING:  WING-TO-WING  (WUC  23000) 


Exhibit  III-8 .  ANOVA  MATRIX  FOR  WUC  23000 


of  our  observations).  This  same  matrix  will  be  used  for  ANOVA 
calculations  later  in  our  analysis. 

The  aggregate  frequency  effects  are  shown  on  Exhibit 
III-9  and  aggregate  training  effects  on  Exhibit  III'-IO.  The 
frequency  effects  show  a  trend  of  increasing  productivity  as 
frequency  goes  from  low  to  high,  although  the  medium-low  and 
medium-high  frequencies  do  not  show  this  throughout.  The  same 
trend  applies  to  training  effects  as  well.  Returning  to  Exhibit 
III-8,  the  specific  effects  are  fairly  evident  since  we  can  see  a 
trend  of  increasing  productivity  as  either  training  or  frequency 
increase.  The  statistical  ANOVA  results  are  discussed  in  Section 
III.B.,  Results. 

3.  Electrical  Power  Supply  (WUC  42000) 

The  productivity  data  by  wing  (base)  for  WUC  42000  and 
its  related  AFSC  423X0  has  been  aggregated  by  action  code  and  is 
shown  on  Exhibit  III-ll.  The  plotted  data  has  been  connected  so 
as  to  show  any  trends  that  might  be  evident  on  Exhibit  III-12. 

Training  effects  are  evident.  Although  training  status 
differences  are  not  large  there  is  a  trend  apparent  of  higher 
trained  wings  having  performed  more  productively  in  most 
instances.  Exhibit  III-13  combines  training  and  frequency 
effects  with  productivity  data.  This  matrix  combines  all  action 
code  productivity  data  and  displays  training  and  frequency  as 
general  groupings  of  low  to  high.  This  same  matrix  will  be  used 
for  ANOVA  calculations  later  in  our  analysis. 

The  aggregate  frequency  effects  are  shown  on  Exhibit 
III-14  and  aggregate  training  effects  on  Exhibit  111-15.  The 


Exhibit  III-9.  AGGREGATE  FREQUENCY  EFFECTS  (WUC  23000) 
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Exhibit  III-10.  AGGREGATE  TRAINING  EFFECTS  (VUC  23000) 
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Exhibit  III-14 .  AGGREGATE  FREQUENCY  EFFECTS  (WUC  42000) 


frequency  effects  are  not  consistent  viewed  from  low  to  high 
although  a  positive  trend  is  generally  evident.  The  training 
effects  are  more  apparent  with  a  fairly  consistent  trend  of 
increasing  productivity  shown  between  low  and  high  trained.  The 
statistical  ANOVA  results  are  discussed  in  Section  III.B., 
Results. 

B.  RESULTS 

In  this  section  we  will  discuss  the  results  we  found  in 
examining  WUCs  14000,  23000,  and  42000  across  wings; 
specifically,  by  the  application  of  ANOVA  and  the  resulting 
statistics.  Also,  we  will  discuss  other  results  we  found  by 
comparing  statistics  for  Luke  AFB  and  Nellis  AFB . 

1.  Overall  Results 

In  our  present  analysis,  we  have  observed  that  varying 
frequency  levels  and  different  levels  of  training  lead  to 
different  productivity.  In  particular,  we  have  observed  that  the 
trend  as  either  training  percentage  and  frequency  increase  is  for 
productivity  to  increase — fewer  workhours  used  per  maintenance 
action.  Although  this  trend  is  observed  for  the  data  that  was 
shown  in  the  previous  section,  we  are  more  concerned  with  the 
generalized  result;  is  this  trend  one  that  we  would  expect  to  see 
throughout  the  Air  Force. 

In  Section  A,  we  developed  ANOVA  matrices  for  each  of 
the  WUCs  examined.  Although  each  specific  action  code  represents 
a  very  different  maintenance  job,  even  within  the  same  WUC,  the 


ANOVA  matrices  shown  previously  include  all  of  the  action  codes 
included  in  this  analysis  for  each  WUC.  This  was  done  for  two 
reasons.  First,  some  action  codes  do  not  include  enough  work 
center  observations  to  adequately  represent  a  sample  population 
to  which  an  ANOVA  analysis  could  be  applied.  Secondly,  the 
rationale  for  accomplishing  this  kind  of  statistical  analysis  is 
to  make  some  kind  of  inference  about  the  general  relationship 
that  both  training  and  frequency  have  to  productivity.  The 
inclusion  of  all  action  codes  makes  this  a  more  general  analysis. 

Averages  computed  for  each  cell  in  the  matrices  can  be 
combined  to  illustrate  what  are  known  as  the  "specific  effects" 
of  both  training  and  frequency.  These  effects  illustrate  the 
change  in  the  average  productivity  value  as  either  training  or 
frequency  change.  Because  cell  averages  include  all  action  codes 
for  each  WUC  in  our  analysis,  the  specific  effects  of  training 
and  frequency  for  WUCs  provide  an  indication  of  productivity 
trends  as  either  frequency  or  training  change  by  themselves. 

Exhibits  111-16,  111-17,  and  111-18  summarize  the 
statistics  developed  for  WUCs  14000,  23000,  and  42000.  The 
pertinent  comparative  statistics  in  these  tables  are  F-statistics 
for  training  (Fj)  and  frequency  (Fp).  Comparing  these  statistics 
with  standard  F-values,  derived  from  the  F-d istr ibut ion  at  a 
specific  level  of  confidence,  allows  us  to  draw  conclusions  as  to 
the  statistical  validity  of  our  results.  Specifically,  if  the 
F-statistics  shown  on  Exhibits  111-16,  111-17,  and  111-18  are 
larger  than  the  appropriate  F-values  from  the  standard  F-distri- 
bution,  then  we  can  conclude  that  the  relationships  we  observed 


DEGREES  OF 

VARIATION  FREEDOM  (D.F.)  MEAN  SQUARES  F-STATISTIC  F-VALUE 


represent  Air  Force  maintenance  and  training  in  general:  training 
increases  productivity.  However,  in  order  to  find  similar 
results  at  other  bases  for  which  we  do  not  have  data  there  should 
be  no  significant  changes  in  work  load,  management  procedures, 
etc.  Additional  research  with  other  types  of  training  and  main¬ 
tenance  should  be  checked. 

The  appropriate  test  results  for  each  WUC  training  and 
frequency  variables  are  provided  below: 

14000 

•  Training 

-  F-statistic  (training)  is  1.09; 

F-value  (from  F-distribution,  90%  confidence,  2 
degrees  of  freedom  (d.f.)  for  training  variance,  4 
d.f.  for  error  variance  is  4.32; 

1.09  is  less  than  4.32,  so  the  relationship 
between  training  and  productivity  in  WUC  14000 
does  not  indicate,  at  the  90%  confidence  level, 
that  training  has  a  positive  impact  on 
productivity;  and 

at  the  75%  confidence  level  the  F-value  is  2.0  so 
we  see  that  training  has  a  positive  impact  only  at 
a  very  low  level  of  confidence. 

•  Frequency 

F-statistic  (frequency)  *  1.97; 

.  -  F-value  (from  F-distribution,  90%  confidence,  2 

d.f.  for  frequency  variance,  4  d.f.  for  error 
variance)  «  4.32; 

1.97  is  less  than  4.32,  so  the  relationship 
between  frequency  and  productivity  in  WUC  14000 
does  not  indicate,  at  the  90%  confidence  level, 
that  Frequency  has  a  positive  impact  on 
productivity;  and 

at  the  75%  confidence  level  the  F-value  is  2.0  so 
we  see  that  frequency  does  have  a  positive  impact 
at  this  lower  level  of  confidence. 
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23000 


•  Training 

F-statistic  (training)  =  6.73; 

F-value  (from  F-distr ibution,  90%  confidence,  2 
d.f.  for  training  variance,  6  d.f.  for  error 
variance)  *  3.46; 

6.73  is  greater  than  3.46,  so  the  relationship 
between  training  and  productivity  in  WUC  230U0 
does  indicate,  at  the  90%  confidence  level,  that 
training  has  a  positive  impact  on  productivity; 
and 

at  the  95%  confidence  level  the  F-value  is  5.14  so 
we  see  that  training  has  a  positive  impact  at  a 
fairly  high  level  of  confidence. 

•  Frequency 

F-statistic  (frequency)  *  .64; 

F-value  (from  F-distr ibution,  90%  confidence,  3 
d.f.  for  frequency  variance,  6  d.f.  for  error 
variance)  *  3.29; 

-  .64  is  less  than  3.29,  so  the  relationship  between 

frequency  and  productivity  in  WUC  23000  does  not 
indicate,  at  the  90%  confidence  level,  that 
frequency  has  a  positive  impact  on  productivity; 
and 

frequency  appears  to  have  a  positive  impact  only 
at  a  very  low  level  of  confidence. 


42000 

•  Training 

F-statistic  (training)  *  6.35; 

-  F-value  (from  F-distr ibution,  90%  confidence,  2 
d.f.  for  training  variance,  4  d.f.  for  error 
variance)  *  4.32; 

-  6.35  is  greater  than  4.32,  so  the  relationship 
between  training  and  productivity  in  WUC  42000 
does  indicate,  at  the  90%  confidence  level,  that 
training  has  a  positive  impact  on  productivity; 
and 


at  the  95%  confidence  level  the  F-value  is  6.94  so 
we  see  that  training  has  a  positive  impact  at  a 
fairly  high  level  of  confidence. 

•  Frequency 

-  F-statistic  (frequency)  *  .43; 

F-value  (from  F-distr ibution,  90%  confidence,  2 
d.f.  for  frequency  variance,  4  d.f.  for  error 
variance)  ■  4.32; 

-  .43  is  less  than  4.32,  so  the  relationship  between 
frequency  and  productivity  in  WUC  42000  does  not 
indicate,  at  the  90%  confidence  levelT  that 
frequency  has  a  positive  impact  on  productivity; 
and 

frequency  appears  to  have  a  positive  impact  only 
at  a  very  low  level  of  confidence. 

2.  Other  Results 


We  undertook  a  comparison  of  Luke  AFB  and  Nellis  AFB  so 
that  we  could  examine  what  effect  simulators  (SAMTs)  had  on 
productivity  as  well  as  the  savings  simulators  provided.  First, 
we  combined  the  WUC  data  for  Luke  AFB  and  Nellis  AFB  for  all  WUCs 
we  looked  at  (14000,  23000,  42000).  This  combined  productivity 
data,  with  training  and  frequency  effects,  is  shown  on  the  ANOVA 
matrix  on  Exhibit  III-19.  It  is  evident  from  this  display  that 
Luke  which  had  a  higher  training  percentage  than  Nellis  is  also 
slightly  more  productive.  Luke  was  89  percent  trained  and  Nellis 
was  66  percent  trained  overall. 

Next,  we  calculated  ANOVA  statistics.  The  statistics 
we  used  are  summarized  on  Exhibit  111-20,  the  ANOVA  table  for 
Luke  and  Nellis  comparison.  We  found  the  following  results  for 
the  training  and  frequency  variables; 
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Exhibit  III-19 .  ANOVA  MATRIX  FOR  LUKE  AFB  AND  NELLIS  AFB 

FOR  COMBINED  WUCS 


DEGREES  OF 

VARIATION  FREEDOM  (D.F.)  MEAN  SQUARES  ^-STATISTIC  F-VALUE 


Training 


-  F-statistic  (training)  *  7.02; 

-  F-value  *  8.53  (90%  confidence);  2.57  (75% 

confidence ) 

-  7.02  is  less  than  8.53  but  greater  than  2.57  thus 
we  do  not  find  a  high  level  of  confidence  in  the 
impact  of  training  on  productivity  comparing  the 
two  bases. 

•  Frequency 

-  F-statistic  (frequency)  *  1.57; 

-  F-value  *  9.0  (90%  confidence,  3.0  (75% 

confidence ) ; 

-  1.57  is  less  than  9.0  (and  less  than  3.0)  thus  we 
find  a  very  low  level  of  confidence  of  the  impact 
of  frequency  comparing  the  two  bases. 

Thus  Luke,  which  is  23  percent  better  trained  than 
Nellis,  and  produced  higher  productivity  results  was  not 

significantly  better  in  a  statistical  sense.  However, 

considering  that  SAMTs  were  used  at  Luke  and  not  used  at  Nellis, 
we  find  that  no  negative  effect  is  apparent  from  the  use  of 
simulators,  rather  than  real  aircraft,  for  maintenance  training. 

In  the  last  section  we  provided  the  overall  cost  of  a 
SAMT  set,  that  supports  a  wing,  as  $1,674,276  in  FY83  dollars. 
This  SAMT  allows  a  supported  wing  to  provide  one-half  less 
aircraft  per  month  on  a  continuing  basis,  which  is  one-half  of 
$18,788,333  or  $9,394,167.  Thus,  the  SAMTs  avoid  $7,719,891  in 
costs  per  wing  supported. 

C.  OBSERVATIONS 

Our  overall  intention  of  showing  a  relationship  of  training 
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to  maintenance  productivity  was  successful  and  provides 
quantified  data  to  support  the  conclusion  that  installation  level 
training  increases  productivity.  The  graphic  representations  of 
frequency  and  training  versus  productivity  lead  us  to  the 
conclusion  that  there  is  a  positive  relationship  between  training 
and  productivity  at  the  installation  level.  The  statistical 
results  using  ANOVA  indicate  that  for  WUC  23000  and  its  related 
AFSC,  as  well  as  for  WUC  42000  and  its  related  AFSC,  our  data  is 
representative  of  the  overall  Air  Force  maintenance  population  at 
the  90  percent  confidence  level  or  higher.  This  is  not  true  of 
our  ANOVA  result  for  WUC  14000,  and  its  related  AFSC,  although  we 
did  find  a  positive  relationship  existed.  The  results  mean  that 
similar  results  would  be  found  at  other  bases  for  which  data  was 
not  collected;  provided  there  are  no  significant  changes  in  work 
load,  management  procedures,  and  so  forth.  Additional  research 
will  see  if  these  results  are  valid. 

We  found  in  our  comparison  of  the  two  wings  located  at  Luke 
AFB  and  Nellis  AFB  that  the  wing  that  was  trained  using  SAMTs  at 
Luke  performed  maintenance  as  productively  as  the  wing  at  Nellis 
that  was  trained  without  SAMTs.  This  was  accomplished  with  a 
savings  of  approximately  one-half  aircraft  assigned  to  support 
training  throughout  the  period  observed.  The  cost  savings 
encountered  here  lead  to  the  conclusion  that  SAMTs  save  money 
and  improve  readiness  by  allowing  additional  assigned  aircraft  to 
be  used  for  operational  purposes. 

It  is  apparent  from  our  expanded  research  in  this  phase  that 
the  interim  results  previously  experienced  have  been  reinforced 
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by  our  current  analysis.  It  is  shown  that  training  does 
directly  influence  maintenance  and  that  delaying  the  training  of 
new  maintenance  personnel  is  detrimental  to  productivity.  The 
wing  analysis  and  training  sections  are  not  presently  attempting 
this  linkage  of  training  and  maintenance  since  their  primary 
concerns  are  to  improve  operational  flying  capability.  Thus, 
statistics  or  data  are  maintained  and  training  is  conducted  but 
the  two  are  not  being  correlated.  If  wing  headquarters  personnel 
were  able  to  do  analyses  similar  to  ours,  they  should  be  able 
to  show  where  training  needs  reinforcement  so  as  to  positively 
affect  maintenance. 

Although  it  was  not  stressed  during  our  previous  discussion, 
there  are  some  maintenance  data  collection  improvements  that 
could  benefit  the  current  system  and  assist  analysis  at  the  wing 
level.  The  first  would  be  the  entry  of  employee  number  for  all 
maintenance  crew  personnel  on  the  AFTO  Form  349  for  entry  in  the 
MDC  system.  This  would  allow  direct  and  precise  measurements  of 
training  and  productivity  for  individuals  and  work  centers. 
Assurance  that  the  use  of  employee  number  does  not  interfere  with 
individual  privacy  rights  must  be  considered.  It  appears  it  does 
not.  Legal  advice  should  be  sought  before  implementation. 

A  second  improvement  would  be  a  greater  use  of  component 
level  WUCs  for  maintenance  reporting  rather  than  system  or 
subsystem  WUCs.  This  would  allow  more  refined  analysis  to  be 
performed  and  would  more  precisely  define  what  work  is  being 
done.  Finally,  the  current  set  of  action  taken  codes  does  not 
sufficiently  define  what  maintenance  tasks  are  being  performed. 
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This  is  only  pertinent  when  applied  to  higher  level*  system  or 
subsystem*  WUCs.  If  component  level  WUCs  were  always  used  then 
the  current  action  taken  codes  would  prove  to  be  adequate. 
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TAC  HQ:  Lanaley  AFB,  VA 

Lt  Col  Kent  Burns 

Air  Force  Logistics  Command: 
Wright-Patterson  AFB, OH 

Mr.  Charles  Cook 

56th  Tactical  Training  Wing: 

TAC/LGQT 

AFLC/MMEMG 

MacDill  AFB,  FL 

(AV  432-XXXX )  or 
(804)  764-XXXX 
3688 

(AV  787-XXXX )  or 
(513)  257-XXXX 

4896 

(AV  968-XXXX)  or 

MSgt  Harris 

MAT 

(813)  830-XXXX 
3212 

Sgt  Stetler 

Analysis 

4708 

58th  Tactical  Training  Wing: 

Luke  AFB,  AZ 

(AV  853-XXXX )  or 

TSgt  Howard  Schultz 

MAT 

(602)  856-XXXX 
3228 

CMSgt  Eliezer  Levine 

FTD  527 

7148 

MSgt  Arthur  Kassel 

Analysis 

6759 

MSgt  Gormiller 

310th  AMU 

- - 

Sgt  Gurney 

310th  AMU 

— 

MSgt  Magee 

311th  AMU 

— 

MSgt  Siperek 

311th  AMU 

— 

363rd  Tactical  Fighter  Wing: 

Shaw  AFB,  SC 

(AV  965-XXXX )  or 

TSgt  Michael  Shaulis 

MAT 

(803)  787-XXXX 
3188 

388th  Tactical  Fighter  Wing: 

Hill  AFB,  UT 

(AV  458-XXXX)  or 

Lt  Col  Carl  Bayha 

MAT 

(801)  777-XXXX 
3835 

Sgt  Bigham 

Analysis 

3529 

474th  Tactical  Fighter  Wing: 

Nellis  AFB,  NV 

(AV  682-XXXX )  or 

Maj  David  Crews 

MAT 

(702)  643-XXXX 
2213 

TSgt  David  Kurau 

MAT 

2216 

MSgt  Gary  Hollis 

Analysis 

3108 

MSgt  Alfred  Siqueido 

FTD  523 

2670 

CMSgt  Klein 

428th  AMU 

— 

Sgt  Green 

428th  AMU 

— 

CMSgt  Anderson 

430th  AMU 

4648 

TSgt  Medina 

430th  AMU 

4648 
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Dr.  Jesse  Orlansky 
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APPENDIX  B 


This  Appendix  contains: 

•  Productivity  and  frequency  data,  and 

•  Analysis  of  variance  technique. 


ANALYSIS  OF  VARIANCE  (ANOVA)  TECHNIQUE 


Step  1 : 

Assign  all  work  center  observations  (training/frequency/ 
productivity  combinations)  to  a  "cell"  in  an  ANOVA  matrix. 
Training  and  frequency  are  the  independent  variables.  This 
examination  attempts  to  explain  the  statistical  weight  of  their 
impact  on  the  dependent  variable  (productivity) . 


^^■^FREQUENCY 

TRAINING 

LOW 

MEDIUM 

HIGH 

ROW 

(TRAINING) 

TOTAL 

LOW 

MEDIUM 

HIGH 

COLUMN 

(FREQUENCY) 

TOTAL 

OVERALL 

TOTAL 

Step  2: 

Once  all  work  center  observations  are  classified  in  the 

appropriate  cell  in  an  ANOVA  matrix,  a  "cell  average" 

productivity  figure  is  computed  in  the  following  manner: 

CELL  AVERAGE  =  All  Productivity  Numbers  for  Work  Centers  in  the  Cell 

Number  of  Work  Centers  in  the  Cell 

This  average  provides  a  measure  of  the  impact  of  a  specific  level 

of  frequency  across  different  training  levels. 


Nuaber  of  Rows)  (Number  of  Coluans) 
2  (Overall  Total)2 


Step  4: 


Calculate  "mean  square"  values  for  the  specific  variations 
computed  in  Step  3.  This  is  a  simple  procedure  where  Vp,  and 
Vp  are  divided  by  their  associated  "degrees  of  freedom"  to  come 
up  with  a  mean  square  value: 

v 

MSj  (Mean  Square  Training)  »  — 

'T  (Degrees  of  Fteedom  for  Training  Variable) 

V 

MSpCmean  Square  Frequency)  -  — 

’F  (Degrees  of  Freedon  for  Frequency  Variable) 

V- 

MSg(Mean  Square  Error)  »  g-j — 

*E  (Degrees  of  Freed  oa  for  Error  Variable) 


Step  5: 

Use  the  "mean  square"  values  calculated  in  Step  4  to 
calculate  an  "F-Statistic"  for  both  training  and  frequency.  The 
F-Statistic  is  a  statistic  that  serves  to  measure  the  explanatory 
power  of  an  independent  variable  on  a  dependent  variable.  The 
statistic  is  computed  in  the  following  form: 

F-Statistic  -  — —  Square  Variance  Attributed  to  «  Specific  Independent  Variable 

Mean  Square  Error  Variance 

A  large  F-Statistic  intuitively  says  that  the  independent 
variable  tested  has  a  much  larger  impact  on  the  dependent 
variable  than  simple  random  change,  or,  in  a  statistical  sense, 
the  independent  variable  is  "significant." 


Once  the  F-Statistics  for  training  and  productivity  have 
been  computed,  the  actual  determination  of  the  significance  of 
those  two  variables  on  productivity  can  be  made.  In  order  to 
perform  this  determination  of  significance,  two  different 
hypotheses  have  to  be  advanced: 

Hq:  Row  (Column)  Average^  -  Row  (Column)  Aver age 2  “  Row  (Column)  Av*rsg«3 

(In  words,  this  hypothesis  states  that  training/frequency  differences  do  not  lead 
to  statistical  differences  In  productivity) . 

and 

Row  (Column)  Average^  4  Row  (Column)  Average^  +  Row  (Column)  Average^ 
(Training/frequency  differences  do  lead  to  atatlstical  differences  in  productivity) . 

To  make  a  determination  as  to  which  hypothesis  should  be 
accepted,  the  F-Statistic  for  both  training  and  frequency 
computed  in  Step  5  should  be  compared  against  a  value  of  the 
F-Statistic  at  a  specific  level  of  confidence.  If  the 
F-Statistic  (computed)  >  F-distribution  (at  a  specific  level  of 
confidence),  then  Hq  is  rejected  and  Hj  is  accepted.  In  effect, 
this  conclusion  says  that  the  independent  variable  being  tested 
(training  or  frequency)  does  have  a  statistically  measurable 
impact  on  productivity  at  a  specific  level  of  confidence. 


ABBREVIATIONS 


APB 

AFR 

AFLCR 

AFSC 

AGS 

AMU 

ANOVA 


Air  Force  Base 
Air  Force  Regulation 

Air  Force  Logistics  Command  Regulation 
Air  Force  Specialty  Code 
Aircraft  Generation  Squadron 
Aircraft  Maintenace  Unit 
Analysis  of  Variance 


CDC  Consolidated  Data  System 

CRS  Component  Repair  Squadron 

CONUS  Continental  United  States 


DCM 

DoD 

EMS 

FTD 

MDC 

MMICS 

MTS 

MI&L 

MRA&L 

OASD 

OSD 

OJT 

RTOK 

SAMT 

SRD 

TAC 

TFW 

TTW 


Deputy  Commander  for  Maintenance 
Department  of  Defense 

Equipment  Maintenance  Squadron 

Field  Training  Detachment 

Maintenance  Data  Collection 
Maintenance  Management  Information  and 
Control  System 
Mobile  Training  Set 

Manpower,  Installations  and  Logistics 
Manpower,  Reserve  Affairs  and  Logistics 

Office  of  the  Assistant  Secretary  of  Defense 
Office  of  the  Secretary  of  Defense 
On-the-Job  Training 

Retest-Okay 

Simulated  Aircraft  Maintenance  Trainer 
Standard  Reporting  Designator 

Tactical  Air  Command 
Tactical  Fighter  Wing 
Tactical  Training  Wing 


WUC 


Work  Unit  Code 


